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Abstract. We present the near-infrared H band lumi- 
nosity function (hereafter LF) of the Coma cluster of 
galaxies. It is the deepest ever computed in the near- 
infrared, for any type of environment, extending over 7 
magnitudes, down to ~ + 6. The LF was computed 
on a near-infrared selected sample of galaxies which pho- 
tometry, complete down to the typical dwarf luminosity, 
is presented in a companion paper. The Coma LF can be 
described by a Schechter function with intermediate slope 
(a ~ —1-3), plus a dip at Mh ~ —22 mag. The shape of 
the Coma LF in H band is quite similar to the one found in 
the B band and, with less confidence, to the R band LF as 
well. The similarity of the LF in the optical and H bands 
implies that in the central region of Coma there is no new 
population of galaxies which is too faint to be observed in 
the optical band (because dust enshrouded, for instance), 
down to the magnitudes of dwarfs. The exponential cut of 
the LF at the bright end is in good agreement with the one 
derived from shallower near-infrared samples of galaxies, 
both in clusters and in the field. This fact is suggestive 
of a similarity of the tip of the mass function of galax- 
ies, irrespective of the environment where they are found. 
The dip at Mh ~ —22 mag is instead unique among all 
the so far measured near-infrared LF, although several 
published observations are not deep enough or spanning 
a suitable wide field to distinctly detect this feature. The 
faint end of the LF, reaching Mh ^ —19 mag (roughly 
Mb ^ —15), is steep, but less than previously suggested 
from shallower near-infrared observations of an adjacent 
region in the Coma cluster. The differences between our 
measured LF and that measured previously in other re- 
gions suggests a dependency on environment of the faint 
end of the mass function (below M* -1-2.5). 

Key words: Galaxies: luminosity function, mass function 
- Galaxies: clusters: individual: Coma (=Abell 1656) 



* Based on observations collected with the Telescope Bernard 
Lyot, at the Pic du Midi Observatory, operated by INSU 
(CNRS). 



1. Introduction 

In the optical band, the cluster luminosity function (here- 
after LF) has three regimes: a bright end {M* ~ —22 mag, 
i/o = 50 km s^^ Mpc^^), a flat slope (a ^ —1.0) down to 
luminosity of dwarf galaxies (Lopez-Cruz et al. 1997; Gar- 
illi, Maccagni & Andreon 1999), and then a steep increase 
(Impey, Bothun & Malin 1988; Ferguson 1989; Thompson 
& Gregory 1993; Seeker & Harris 1996; Seeker, Harris & 
Plummer 1997). Often a dip is found in the otherwise flat 
part of the LF (see, e.g., Godwin & Peach 1977; Buck- 
nell, Godwin & Peach 1979). A few well studied clusters 
(Smith, Driver & Phillipps 1997), as well as number of 
LFs pubHshed a long time ago (Schechter 1976) display 
an intermediate slope (a ~ —1.3) instead of a flat LF. 

The LF represents the zero-order statistics of galaxy 
samples and gives the relative number of galaxies as 
a function of the magnitude. Almost every quantity is, 
therefore, "weighted" by the LF, including obvious quan- 
tities, such as the galaxy color distribution, and also less 
obvious ones, such as correlations involving the luminosity 
(see, for example, the discussion on the impact of magni- 
tude limits in the size-luminosity relation by Simard et 
al. 1999). When the sample is not complete in volume a 
further "weight" should be added: the selection function. 
Thus, an accurate knowledge of the LF is important when 
comparing galaxies of different luminosities at different 
redshifts. 

From a physical point of view, the optical LF is the 
convolution of the number of galaxies of a given mass with 
their M/L distribution. Then, any measure of the optical 
LF traces a complex mix of galaxy mass and M/L dis- 
tributions, so that evolution in luminosity or mass could 
not be easily disentangled from the measurement of the 
optical LF. A better estimate of the galaxy mass than the 
optical luminosity will certainly help to separate the two 
dependencies. Such a measure has a particular relevance 
in the determination of the density of the Universe: a pos- 
sible way to proceed is to compute the cluster mass per 
unit luminosity times the Universe luminosity density. As 
stressed by Calberg et al. (1996), this calculation assumes 
that cluster galaxies have the same LF as field galaxies. 
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Observations suggest instead that galaxies change their 
optical luminosity during their infall in the cluster (see, 
for example, Bothun & Dressier 1986; Andreon 1996, and 
most of the papers by the CNOC collaboration, such as 
Balogh et al. (1998) and references therein), although the 
amplitude and the sign of the luminosity variation is not 
yet settled. Of course, it would be preferable to measure 
the M/L of clusters using a luminosity indicator weakly af- 
fected by possible bursts or halt of star formation induced 
by interactions with the hostile cluster environment. 

The near-infrared luminosity has several advantages 
with respect to optical luminosities. It is tightly corre- 
lated to the galaxy mass (at least for spirals, Gavazzi, 
Pierini & Boselli 1996) and, with respect to the optical 
luminosity, it is less affected by short and recent star for- 
mation events (Bruzual & Chariot 1993), possibly induced 
by interactions, and by dust absorption. Therefore, the 
near-infrared LF traces more directly the mass function 
and gives a Universe density less affected by possible sys- 
tematic errors due to a differential star formation history 
between galaxies in clusters and in the field. 

There are several additional advantages in observing 
galaxies in the near-infrared: K corrections are relatively 
small and well known, thus allowing to observe and to 
compare galaxies at different redshifts, up to high redshift 
values. In particular, K corrections are almost indepen- 
dent from the spectral type of galaxies, in such a way 
that statistics on a population of galaxies are less affected 
by changes of the morphological composition induced by 
differential corrections from type to type. Furthermore, 
galaxies that undergo a starburst are not selected prefer- 
entially, as instead happens in the optical, and therefore 
a sample selection in the near-infrared is less biased by 
episodic events of star formation. 

It is therefore important to measure the near-infrared 
LF of clusters of galaxies over a magnitude range as wide 
as possible, in particular to characterize the properties of 
galaxies in the local Universe. So far, the near-infrared LF 
have been measured for a few clusters, but to bright limit- 
ing magnitudes (Barger et al. 1996, Trentham & Mobasher 
1998, De Propris et al. 1999), and on a portion of the 
Coma cluster (De Propris et al. 1998), down to relatively 
faint magnitudes. According to De Propris et al. (1998), 
the Coma LF shows a flat slope, and a step increase 
(a ~ ~l-7) at faint magnitudes {H = 16 mag). However, 
this is presently the only LF determination attaining inter- 
mediate magnitudes, and such a survey could be improved 
in several respects. It is important to extend the study to 
other regions, and to reach deeper magnitudes. This is the 
aim of the present paper. 

We present the near-infrared LF of the Coma cluster, 
based on independent observations, fully documented in a 
companion paper that also presents the photometric cat- 
alog. With respect to De Propris et al. (1998), this study 
has been performed on a different portion of the Coma 
cluster, slightly overlapping with their one, over an area 



which is ^ 40 % smaller, but it attains one magnitude 
deeper. All along this paper, we adopt Hq = 50 km s~^ 
Mpc^i and qo = 0.1. 

2. Data analysis 

Coma galaxy counts and LF have been computed from 
the photometry presented in Andreon, Pello, Davoust et 
al. (1999), to which we defer for details (hereafter Paper 
I). In summary, a ~ 20 x 24 arcmin region of the Coma 
cluster, located ~ 15 arcmin from the centre, have been 
imaged with the Moicam camera at the 2.0m Bernard Lyot 
telescope at Pic du Midi. Images were taken in the H 
band under moderate to good seeing conditions (i.e. 1 < 
FWHM < 1.5 arcsec), with average exposure time of 
^ 300 sec. About 300 objects have been detected and 
classified by Sextractor version 2 (Bertin & Arnouts 1996) 
in the best exposed part of our mosaic (~ 380 arcmin^). 

Different magnitudes are presented in Paper I. We 
adopt here the Kron magnitudes (see Kron 1980 for the 
exact definition, and Bertin & Arnouts 1996 for the soft- 
ware implementation). They are defined as the flux mea- 
sured in a region which area is adapted to each galaxy. 
Unfortunatly, they depend sensibly on the determination 
of the object size, in particular for faint objects, and there- 
fore, for faint objects we prefer aperture magnitudes. More 
precisely, we adopt, as a measure of the magnitude for a 
galaxy, the magnitude computed within 2.5 Kron radii for 
galaxies brighter than iJ = 14 mag, and aperture magni- 
tudes (within 10 arcsec aperture) for fainter galaxies. The 
two quantities are identical, within the errors, for galaxies 
in a large magnitude range including H ^ lA mag (Pa- 
per I). The catalog is complete, in the 10 arcsec aperture 
magnitude, down to H = 17.1-17.2 magQ. To be safe and 
for easy computation, we cut the catalog at H = 17.0 
mag. Given the galaxy catalog and the knowledge of the 
surveyed area, galaxy counts in the Coma direction are 
computed straightforwardly. They are presented in Figure 
1 (open dots), as derived for objects identified as galaxies 
(see Paper I for details). 

The Coma cluster LF is computed as the statistical 
excess of galaxies in the Coma cluster direction with re- 
spect to other directions. In order to estimate the fore and 
background contribution of the field, we use when possible 
observed values measured by different authors, as well as a 
simple standard model for number counts, based on pure 
luminosity evolution for galaxies (sec Rocca-Volmerange 
& Guiderdoni, 1990, Pozzetti et al. 1996 and Pozzetti et 
al. 1998) and computed through the Bruzual & Chariot 
evolutionary code (1993, updated as GISSEL98). The pa- 
rameters of the counts model have been set up in order 
to roughly reproduce the observed number counts to B = 
28 mag (Williams et al. 1996), and normalized to the ob- 
served counts at H — 17.0 mag. This model is only used 

^ All magnitudes are refered to the Vega system 
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in order to derive the mean redshift of the dominant pop- 
ulation at a given magnitude, when comparing with other 
LF estimates, computed with other filters. 

Field counts have been measured on the H band im- 
ages of the Hubble Deep Field South 1 & 2 (hereafter 
HDFSl-hS2), presented in da Costa et al. (1999). These 
images were taken at the NTT and they are much deeper 
(several magnitudes) than our Coma images, but extend- 
ing to a smaller region and exposed in an non uniform 
way. We have used their uniformly exposed part, a central 
region 1000 x 1000 pixel wide (i.e. ^ 23.7 arcmin^ large, 
thus more than 10 times smaller than the Coma area stud- 
ied in this paper). We have detected and classified objects 
in this HDFS1-I-S2 area by means of Sextractor (Bcrtin & 
Arnouts 1996), using the same parameters as in Paper I. 
Figure 1 presents the resulting counts (closed triangles). 
At ~ 16 mag, field counts have large errors because the 
HDFS1+S2 is not tailored for measuring galaxy coimts at 
such bright magnitudes, but for going deep on a small re- 
gion. This fact prompt us to look for a H band survey more 
adapted to our aims, i.e. shallower and wider. Since it does 
not exist, we get a different estimate of the H band galaxy 
counts using K band galaxy counts and assuming a mean 
H — K color for galaxies in the relevant magnitude range. 
The observed color of ~ 17 mag galaxies is i/ — if ^ 0.6 
mag (Stanford, Eiscnhardt & Dickinson 1995). This value 
is also in fairly good agreement with the mean H — K 
expected from the counts model [H — K ^ 0.55 mag to 

~ 19 mag, where the population is dominated by galax- 
ies with 0.1 < z < 0.4 at iJ ~ 17 mag, and with 0.2 < .z < 0.6 
atil~18 — 19 mag). We apply the mean H — K value to 
the Bershady, Lowenthal & Koo (1998) compilation of K 
band surveys. These counts are presented in Figure 1 as a 
strip with center given by the average counts presented in 
their paper. There are also presented in Figure 1 the ex- 
pected number counts derived from our model (solid line 
histogram). In spite of unavoidable differences between the 
types of magnitude used by the different authors, and also 
the approximations involved in the conversions between 
photometric systems, the agreement between the galaxy 
counts in the HDFS1-I-S2 direction and the H counts esti- 
mated from K counts is very good. They are also in good 
agreement with the counts derived form our simple model. 
We adopt these counts as average background counts in 
the Coma direction. 

An expected and important source of error in the LF 
determination is the background variance from field to 
field, in addition to Poissonian fluctuations: if thc^ back- 
ground variance is high, then the background in tlie Coma 
direction could be significatively different from the average 
computed above. Among the K band shallow surveys, two 
of them are adapted to roughly compute the order of mag- 
nitude of this variance. Gardner et al. (1993) presented 
galaxy counts for the HMDS (Hawaii Medium Deep Sur- 
vey) extending over an area which is only half of that sam- 
pled for Coma, and also for the HMWS (Hawaii Medium 



Wide Survey) , extending over a much wider area than the 
present one. The counts in the two surveys show a ±15 % 
scatter, and we adopt this value as a typical fluctuation for 
the background (to be added quadratically to Poissonian 
fluctuations). The amplitude of the strip in Figure 1 shows 
this scatter. This background variance seems plausible for 
two reasons: first, counts in the HDFS1-I-S2, which extend 
on an area ^ 10 times smaller than our one, and ~ 3 times 
smaller than the HDMS survey, are well within the strip, 
showing that background fluctuations are unlikely to be 
larger than our derived variance. Secondly, the expected 
field to field fiuctuations are ~ 11 %, according to the 
formulas (and the hypothesis) in Huang et al. (1997). 

Figure 1 shows that at all magnitudes considered here, 
the Coma cluster counts have small errors and stand out 
with respect to the field counts, down to iJ = 17 mag. 
Therefore, errors on the Coma LF will be small and only 
slightly affected by the background subtraction. In order 
to judge on the progress achieved in this paper with re- 
spect to previous investigations, the reader can compare 
our magnitude-counts diagram with the analogous one in 
Mobasher & Trentham (1998) for a much smaller (and 
denser) region of Coma. These authors took an obser- 
vational strategy quite different from our one: given the 
available telescope time, they went as deep as possible on 
a very small area, which resulted in a large field to field 
background variance. 

3. Results 

3.1. The shape of the LF 

Given the counts in the Coma direction and in the field, 
and their errors, the computation of the Coma cluster LF 
is an algebrical exercise. We stress that our main sources of 
error on the Coma LF are Poissonian fluctuations of total 
counts over the Coma region and the background field to 
field variance. Therefore, the error on the background is 
not derived from the HDFS1-I-S2 errorbars (which are not 
relevant for the determination of the LF errors). 

The near infrared Coma LF is presented in Figure 2. 
It is characterized by a bright end (at Mh —25 mag), 
a part increasing gently down to Mh ~ —18.5 mag, and 
an "outlier" point at Mh = —22.2 mag, which produce, if 
real, a dip in the Coma LF. The LF displayed in Figure 
2 is the deepest ever measured in any near-infrared band 
for any type of environment for a near-infrared selected 
sample. 

In the optical, the Coma cluster exhibits a similar 
shape (Godwin & Peach 1977; Seeker & Harris 1996). Us- 
ing Godwin, Metcalfe & Peach (1983) data, we computed 
the b (a photographic J-like blue filter) Coma LF in al- 
most exactly the same area surveyed in the H band. For 
simplicity, we have considered a rectangular area enclosing 
our H band region, without taking into account the com- 
plex geometry of our region in details. Then, because the 
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H and b band magnitudes are available for all galaxies in 
this area, we have computed a mean b — H pseudo-color^. 
In order to compare the two LFs, we have shifted the b 
band LF by the mean b — H pseudo-color: b — H = 3.5 
mag. The expected values according to Bruzual & Char- 
lot models are b — H = 4.0 for elliptical galaxies and up 
to b — H = 2.8 for blue constant star-forming systems; 
thus an averaged population of 2/3 of ellipticals versus 
1 /3 of blue systems gives roughly the right mean value as 
expected. No normalization in An has been applied. The 
result of this exercise is shown in Figure 2 as a dashed-line 
histogram. The two LFs are remarkably similar: there is a 
close agreement between the cut at AIh = —25 mag, the 
dip location {Mh = —22.2 mag), the dip amplitude and 
the increase observed at fainter magnitudes (closed dots) 
and expected from the b LF (dashed histogram). We have 
performed the same exercise with the R band photome- 
try by Seeker & Harris (1996), who studied an adjacent 
region of the Coma cluster. The expected values accord- 
ing to Bruzual & Chariot models are R — H = 2.5 for 
early-type galaxies and R — H = 2.0 for blue star-forming 
systems, giving an averaged value of R — H = 2.3 for the 
same weighted population taken above (2/3 of ellipticals 
versus 1 /3 of blue systems) . This time we adopt the pre- 
dicted value because the R catalog is not published. The 
two surveyed regions are different, thus we normalize their 
LF to our H LF. We obtain similar results, with the dif- 
ference that the importance of the dip is smaller in R than 
in H (see Figure 2). 

Thus, the shape and the amplitude of the Coma LF 
seems not to be strongly depend on the wavelength when 
we compare the results in b and in H bands, and also, 
with less confidence, in the R band. The strong similar- 
ity of the optical and near-infrared LF implies that in the 
near-infrared there is no new population of galaxies which 
disappears in the optical band (because dust obscured, for 
example), down to the magnitude of dwarfs. Furthermore, 
if the H band LF traces the galaxy mass function in this 
cluster, the same holds true for the blue LF. This result 
has been obtained in a particular region of Coma, wich is 
a cluster rich in elliptical and lenticular galaxies. Before 
any generalization, this result should be checked in other 
regions of the cluster, and also in other environment con- 
ditions (cluster outskirts, clusters rich in spiral galaxies, 
groups, ...). 

3.2. The dip at Mh 22 

Let us consider in more details the dip point. The ques- 
tion is: Is it really an outlier? The statistical significance 
of the possible outlier point must be evaluated from the 
galaxy counts, since they are the original source of fluctu- 
ations. The Mh = —22.2 mag bin in Figure 2 corresponds 

^ As long as the two magnitudes are not computed within 
the same aperture, the difference is not actually a color. 



to the H = 13.5 mag bin in Figure 1. First of all, we ex- 
clude the possibility that we have missed some galaxies of 
this magnitude, because we are complete 2.5 mag fainter, 
and because a typical galaxy oi H = 13.5 mag have a 
central brightness of 100 times the sky noise. Secondly, 
there is no relation between the location of the dip and 
the discontinuity of the magnitude system adopted (Kron 
magnitudes for bright galaxies and aperture magnitudes 
for faint ones): galaxy counts does not change because the 
separation is set to H — lA mag. In fact, Kron and aper- 
ture magnitudes have almost the same value down to the 
last magnitude bin (see figure 10 in Paper 1). There are 
two galaxies in the H = 13.5 mag bin, whereas ^ 13 are 
needed to make the counts smooth. Therefore, this point 
is more than 3 a away from the average of adjacents bins. 
To be precise, according to Poissonian statistics we can 
reject at more than 99.95 % confidence level the hypoth- 
esis that the observed number of galaxies is drawn from 
a parent distribution which counts ^ 13 galaxies in that 
bin. Therefore, the dip is a real feature of the Coma near- 
infrared LF in this region. 

The dip in the Coma LF had firstly been noticed in 
the optical band (for example, Godwin et al. 1983) and it 
had been interpreted in two different ways. Biviano et al. 
(1995) suggested that galaxies brighter than the dip were 
subjected to a recent episode of star formation induced 
by the hostile Coma environment, which have made them 
brighter. Andreon (1998) has shown that the LF of the dif- 
ferent morphological types of galaxies are equal in Coma 
and in much poorer environments, and that the dip is 
simply the combined result of the Coma cluster morpho- 
logical composition together with the shape of the type- 
dependent LFs. If the induced star- formation interpreta- 
tion by Biviano et al. (1995) was correct, the dip should 
be absent or at least highly attenuated in iJ, because the 
near-infrared luminosity traces the galaxy mass and it is 
less affected by the short timescale starbursts that makes 
a few galaxies brighter than the magnitude of the dip. 
Instead the dip is observed in the H band. 

A few other near-infrared LF of clusters are (poorly) 
known. None of the five clusters studied by Mobasher & 
Trentham (1998) show such a dip. However, the area sam- 
pled in each cluster includes a tiny number of galaxies, so 
that errors are large and the visibility of a possible dip (if 
present) is arguable. The cumulative LF of three clusters 
at 2 ~ 0.3 (Barger et al. 1996) does not seem to show 
a dip, but it barely reaches the dip magnitude. The only 
truly comparable LF has been presented by De Propris 
et al. (1997), and it is reproduced here in Figure 4 (open 
squares), together with the best Schechter fit LF (dotted 
line) to our data. The fitting machinery adopted here is 
discussed in the next subsection. De Propris et al. (1997) 
have used Kron magnitudes for faint galaxies and aper- 
ture magnitudes (within a 62 arcsec diameter) for large 
galaxies (De Propris 1999, private communication). The 
two LFs are in remarkable good agreement (x^ < 1) on 
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the common range {Mu < —20 mag), with the exception 
of the dip bin, wich is present in our data and absent in 
De Propris et al. (1997) data. It is worth to note that the 
position of the dip is well within the spectroscopic sam- 
ple of De Propris and collaborators, and thus it could be 
hardly missed. The agreement would be even better at 
Mh < -24 if the De Propris et al. (1997) bright magni- 
tudes were of Kron type, since Kron magnitudes integrate 
the galaxy flux inside a smaller area than those sampled 
by the 62 arcsec aperture they used. 

Since De Propris et al. (1997) studied an almost com- 
plementary area of the Coma cluster with respect our one, 
and the dip is present in our LP and absent in theirs, 
it is possible that the amplitude of the dip depends on 
the location in the cluster, as it seems to be the case in 
the optical (Sekiguchi 1998). Since the H band luminosity 
traces the galaxy mass, as stressed in the Introduction, 
the possible dependence of the dip amplitude on the clus- 
ter location points out a dependence of the mass function 
on the surveyed region, possibly due to a joint effect of 
morphological dependence of the LF and variation of the 
morphological composition over the Coma cluster. A sim- 
ilar trend is seen in the optical (Andreon 1998). Such dif- 
ferences in the LF as a function of the location in the clus- 
ter could be related to subcluster structure. Several evi- 
dences for cluster-cluster merger are present in Coma. Two 
main peaks appear in the X-ray flux density (White et al. 
1993), in the projected distribution of galaxies (Fitchett & 
Webster 1987, Mcllicr ct al. 1988) and in the radio source 
counts (Kim et al. 1994): a clump centered on NGC4874 
and NGC 4889, and a secondary peak around NGC 4839, 
about 40' SW from the previous one. The field surveyed 
here is centered ~ 15' NE from the main structure, at the 
opposite side with respect to the cluster center. Colless et 
al. (1993) have shown the complex dynamics and multi- 
ple substructure of the Coma cluster using a large redshift 
catalog. According to them, the NGC 4839 group is actu- 
ally falling into the main cluster, there are two subclusters 
in the central region (associated with the two dominant 
galaxies), and late type galaxies are falling into the main 
cluster (which is dominated by early type galaxies). These 
processes might be able to locally modify the LF as ob- 
served. 

3.3. Fitting the LF 

Let us consider now the overall shape of the LF. Usually, a 
method is used to fit the LF of clusters by a Schechter 
(1976) function: 

/(m) =(j)* io0-4(a-n)(™*-m) expi-lO^ *^""' ~"'^) 
The method is not the optimal one for fitting a 
function to a small number of bins, and it is even less 
suitable when bins are poorly populated. Furthermore, a 
requires to bin the data with an arbitrary bin size. Al- 
though the method is not optimal, we are forced to use 
it, since we do not know any other fitting method that 



could take into account, even roughly, background fluctu- 
ations together with Poissonian ones without binning the 
data. More elegant methods implemented so far, such as 
maximum-likelyhood fitting, do not take into account nei- 
ther Poissonian fluctuations of the background counts, nor 
the field to field variance of the background, and therefore 
they systematically underestimate the true errors. 

In order to take into account the amplitude of the bin 
in the fitting process (a technical detail seldom consid- 
ered) , we fit the data with a Schechter function convolved 
with the bin width (although in practice this detail makes 
almost no difference on the results). An additional prob- 
lem arises: given the existence of a real dip in the Coma 
LF, the fit of the whole LF with any Schechter function 
is necessarily poor (and in fact we found a minimum x'^ 
of 14 for 4 degrees of freedom). We are therefore left with 
two options: flag the dip point, or use a more complex 
function. Disposing of a very small number of points and 
lacking any physically motivated more elaborate function 
to be fitted, we simply flag the outlier bin. 

In that case, and taking into account the finite ampli- 
tude of the bin, we found M|j ~ —24.6 mag and a — —1.3, 
but with large confidence intervals (as shown in Figure 3). 
Note that our magnitude limit, H = 17 mag, is roughly 
equivalent to Mb ~ — 15 mag at tlie Coma distance for an 
early-type galaxy {B — H 4 mag) , which is well in the 
dwarf regime. M* agrees well with the values expected 
from the optical photometry and usual colors for early- 
type galaxies {Mb = —20.5 mag and B — H ^ 4 mag). 
The slope is steeper than the typical value in optical bands 
(Lopez-Cruz et al. 1997; Garilli, Maccagni & Andreon 
1999), but nevertheless it is quite similar to that found 
for a few well studied clusters in optical bands (Smith, 
Driver & Phillipps 1997; Schechter 1976). 

3.4- Comparison to previous studies of Coma and to the 

field LF 

Mobasher & Trentham (1998) studied a very small por- 
tion of the Coma cluster and were able to build a catalog 
1.5 magnitudes deeper than ours. However, their studied 
field is too small to make the background variance small 
relative to the signal (the Coma LF), so that the result- 
ing LF is completely unconstrained, as admitted by the 
authors. They computed also another LF, by performing 
a crude color selection, i.e. assuming that Coma cluster 
galaxies lay, in a color-color plane, in a region different 
from that occupied by the fore and background galaxies. 
In that LF with errorbars of reasonable size was 

derived, but under an hypothesis that should be demon- 
strated to be true. In Figure 5, this LF is plotted over- 
lapped to our H LF, after having matched the two LFs in 
the common bins. Their K magnitudes have been changed 
to H assuming H — K = 0.24 mag, the typical value ex- 
pected for the Coma galaxies. Our errorbars are smaller 
in the common bins, even using the same binning for the 
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two LFs. Mobasher & Trentham (1998) points stay rela- 
tively near the extrapolation of the best Schechter fit to 
our data, suggesting that the LF could keep its a ~ —1.3 
slope even at these very faint magnitudes (roughly equiv- 
alent to Mb ^ —13.5 mag). The same points stay near 
the Seeker & Harris (1996) R band Coma LF shifted in 
the H band, as plotted in Figure 2. 

Figure 5 also compares the Coma cluster LF to the lo- 
cal field LFs, as computed by Gardner et al. (1997, dashed 
line) and Szokoly et al. (1998, solid line). The two field 
surveys differ in many respects. The former is based on a 
sample about 5 times larger than the latter, and it is com- 
puted from a near-infrared selected sample. Instead, the 
latter is optically selected and no corrections have been 
applied for the optical selection. The slope of the field LF 
computed by Gardner et al. (1997) and by Szokoly et al. 
(1998) differ largely, with a ^ —0.9 and a ~ —1.3, respec- 
tively. However, the 68 % confidence contours of the two 
LFs cross each other (figure not shown), implying that 
the two LFs are compatible to ~ Itr, as also claimed by 
Szokoly et al. (1998). The two field LFs could have differ- 
ent slopes but they still remain compatible because they 
barely reach Mh — —21.5 mag^, and therefore they sam- 
ple only the exponentially declining part of the LF. There- 
fore, the slope of the field LF is constrained by the faintest 
bin (see Figure 5) which, as in all field surveys, is quite 
uncertain because measured on a very small volume. 

Our own data for the Coma H LF are three mag 
deeper than the local field ones, and the overall shape, 
as parametrized by the Schecther parameters, agrees with 
the field ones: the Coma LF has a and M* indistinguish- 
able from the Szokoly et al. (1998) LF and a M* very sim- 
ilar to the Gardner et al. (1997) one. Its slope, a — —1.3, 
is steeper than the Gardner et al. (1997) slope, but by less 
than ~ 1(7 difference, due to the large confidence level in- 
tervals of the two LFs. Also, the present Coma H band LF 
agrees with the field LF computed by Cowie et al. (1996) 
in a few redshift ranges, up to z 1. 

On one side, the overall shape of the LF is similar 
both in the field and in the Coma cluster. On the other 
side, no dip is present in near-infrared field LFs, whereas 
instead in our Coma LF it is quite evident. This fact, 
and the absence of the dip in the Coma region studied 
by De Propris et al. (1998) seems to suggest that the dip 
amplitude could be related to the morphological mix of the 
studied environment. The alternative possibility requires 
that environmental effects change the H band luminosity 
preferentially at a given mass (corresponding to iJ ~ —22 
mag), without altering too much the mass distribution for 



^ Both LsF are originally computed in the K band, and have 
been transformed in the H band assuming the same mean rest- 
frame color for all galaxies [H — K = 0.2 mag). This is a rea- 
sonable value for the brightest population, which is dominated 
by galaxies at z < 0.1 



more massive galaxies. Otherwise, the Coma and the field 
LFs should have different bright tails. 

4. Discussion & Conclusions 

We have presented the near-infrared LF of a nearby clus- 
ter of galaxies, Coma, down to faint magnitudes {Mh = 
— 18.5 mag, i.e. M^ + 6 mag corresponding roughly to 
Mb ^ —14.5 mag). This has been provided to be possible 
due to the relatively deepness of the present images and to 
the small background variance associated with the large 
surveyed area. The computed LF is the deepest ever mea- 
sured in the near-infrared, on any type of environment. 

The shape of the Coma LF in the region studied seems 
not to depend on wavelength, at least in b and H bands, 
and with less confidence, in R. The similarity of the LF 
implies that in the central region of Coma there is basi- 
cally no new population of galaxies which disappears or 
becomes too faint to be observed in the optical bands 
(because of the presence of dust, for instance), down to 
the magnitudes of dwarfs. Furthermore, if the H band LF 
traces the galaxy mass function, also the blue LF traces 
the mass in this case. This is in apparent contradiction 
with the results by Gavazzi, Pierini & Boselli (1996), who 
found that for spiral galaxies the M/L is approximatively 
constant in the near-infrared but not in the optical filters. 
Since our finding is based on just one sample in one partic- 
ular environment, although selected with well understood 
selection criteria (volume-complete), it has to be verified 
on other samples of nearby galaxies, possibly spiral-rich 
clusters or groups, before any dangerous generalization. 

The bright part of the Coma H band LF, i.e. the 
brightest three magnitudes, agrees with the expectations 
based on optical LFs and usual colors for galaxies, and 
with what is observed in shallower near-infrared surveys 
of clusters of galaxies and also on the field. This con- 
firms that the shape of the tip of the mass function seems 
environment-independent and therefore environmental ef- 
fects have a minor impact on the luminosity of bright 
galaxies (M < M* + 2), and possibly on their masses. 
Coma and the field population differ by a factor of 100 
in galaxy density. The extension of this sentence to faint 
{M > M* + 4) galaxies still await a determination of the 
field LF in the dwarf regime. 

The Coma near-infrared LF presents a real dip at a 
luminosity corresponding to that observed in the optical 
LF. This is the first detection of such a feature in the 
near-infrared. The existence of a dip in the Coma LF in 
the H band implies the presence of a dip also in the galaxy 
mass function. To our knowledge, there is presently no 
simulation of cluster formation which is able to produce 
such a feature in the galaxy mass or luminosity function. 
This feature, being distinctive, will set a strong constraint 
for the future simulations. 

Kauffmann & Chariot (1998) have shown that the 
apparent passive evolution and the slope of the color- 
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magnitude relation can be accomodated within a hier- 
archical model, even if the galaxies themselves grow by 
mergers until late times. One of the important remaining 
issues is the comparison between the predicted and the ob- 
served LF, in particular, the distribution of galaxies as a 
function of their morphological type, at lest for early-type 
galaxies. Probably the main limitation till now has been 
the lack of suitable observational data to compare with 
model expectations. Our near-infrared catalog, published 
in Paper I, joint to the morphological types for the Coma 
galaxies, available from Andreon et al. (1996, 1997), fill 
this observational gap. 

The overall slope of the Coma LF is intermediate 
(a ~ —1.3). The slope is measured down to the dwarfs 
regime: wc reach Mb ^ —14.5 using own our data alone 
and even fainter magnitudes {Mh ~ —17, roughly equiva- 
lent to Mb ~ —13) when including Mobasher & Trentham 
(1998) data and under their assumptions. When compar- 
ing Coma and the field LFs in the near-infrared, we have 
to take into account that both LFs have been derived with 
completely different data and methods, because of the dif- 
ferent selection criteria for the two samples: the field LF 
is computed on a flux-limited sample, whereas the cluster 
LF is computed on a volume-limited sample. In particular, 
the field LF suffers from a 10 % redshift incompleteness 
(or is based on an optical selection, as for the Szokoly et al. 
(1998) LF), and a poor sampling of faint luminosities, be- 
cause of the small volume explored at that luminosities. 
Nevertheless, and even if the environments sampled are 
quite different, the bright tail of the Coma and the field 
LFs are in close agreement. In our opinion, this exclude 
the possibility of large systematic errors in the derivation 
of field LFs, and therefore indirectly confirms the disagree- 
ment between the observed near-infrared LF and that ex- 
pected on theoretical grounds in the present simulations 
of a hyerarchical Univcrs (Kauffmann, Colbcrg, Diafcrio 
& White, 1999). This also suggests that more detailed 
models are needed to reproduce the observed properties of 
galaxies, such as the LF. In particidar, as mentioned be- 
fore, the existence of a dip in the present LF and the large 
range on which this LF is computed (7 mag) provides a 
strong constraint to future simulations. It is worth to note 
that theoretical predictions on the behaviour of high order 
statistics, such as the color distribution or the galaxy evo- 
lution, use a particular realization of the LF as a "weight" . 
Thus, increasing the accuracy on the determination of the 
LF will certainly contribute to the improvement of the 
theoretical knowledge on galaxy formation and evolution. 
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Fig. 1. Galaxy counts as a function of the apparent H 
magnitude. Open dots are the counts in the Coma cluster 
direction, solid triangles are counts in the HDF South 1 & 
2 directions. The center of the strip marks the average K 
counts converted in H assuming H — K ^ 0.6. The strip 
width correspond to a background variance of ±15%, the 
typical value for the area surveyed in Coma. The solid 
line histogram gives for comparison the expected counts 
from our model. See text for details. Errorbars are com- 
puted according to Geherls (1986). Bins are 1 magnitude 
wide. The abscissa is given by Kron magnitudes for bright 
galaxies and aperture magnitudes for faint galaxies. 



100 




S. Andreon, R. Pello: H band Coma luminosity function 

16 18 20 



Fig. 2. Coma LF in the H band, as computed from the 
present data alone (closed dots). The solid line is the R 
Coma LF, shifted by a color R—H = 2.3 mag. The dashed 
histogram is the b Coma LF, shifted by a pseudo-color 
b — H = 3.5 mag. The dotted line is the best fit by a 
Schechter function, once the dip point is flagged. Errorbars 
in the ordinate axis are computed according to Geherls 
(1986) and include also the field to field variance of the 
background. Errorbars in the abscissa show the bin width. 
Errorbars for the histogram are similar to those of points. 
The upper abscissa scale shows the apparent H magni- 
tude, and the lower one gives the corresponding absolute 
H magnitude. An is the number of Coma galaxies in the 
studied field. 
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Fig. 3. 68 % and 95 % confidence contours for the fit of 
the LF by a Schechter function. The units of the left and 
right ordinates are absolute and apparent H magnitudes, 
respectively. 
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Fig. 4. De Propris et al. (1998) LF (open squares) com- 
pared to the best Schechter fit to the present data (dotted 
line). Errorbars and scales are as described in Figure 2. 
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Fig. 5. Various determinations of the near-infrared LF. 
Our own data (solid dots) and Mobasher & Trentham 
(1998) data (open squares) are shown, after normaliza- 
tion of the LF in common bins. For details on the deriva- 
tion of the Trentham & Mobasher (1998) data points, see 
the text. The dotted curve is the best fit of our data, ex- 
trapolated to fainter magnitudes. Local field LFs are also 
shown: Gardner et al. (1997) (dashed line) and Szokoly 
et al. (1998) (solid line). The field LF has been vertically 
shifted to reproduce the Coma LF in the three brightest 
bins. Error bars and scales are as described in Figure 2. 



